{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "20fd4f21",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/pinecone-io/examples/blob/master/learn/search/multi-modal/recipe-search/recipe-search.ipynb) [![Open nbviewer](https://raw.githubusercontent.com/pinecone-io/examples/master/assets/nbviewer-shield.svg)](https://nbviewer.org/github/pinecone-io/examples/blob/master/learn/search/multi-modal/recipe-search/recipe-search.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3adc417",
   "metadata": {},
   "source": [
    "<h1>Inverse recipe search</h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24dab2d3",
   "metadata": {},
   "source": [
    "This notebook demonstrates how Pinecone can handle very large embeddings that come from a sophisticated model trained using a large multimodal food dataset. The paper is available at this link: https://research.fb.com/wp-content/uploads/2019/05/Inverse-Cooking-Recipe-Generation-from-Food-Images.pdf.\n",
    "\n",
    "We have built this recipe search by image using Pinecone in the following way:\n",
    "<br>\n",
    "- From Facebook's large model, we have extracted 512 x 7 x 7 large embeddings from their convolutional network\n",
    "<br>\n",
    "- We have used these features to obtain ingredients and instructions vocabularies\n",
    "<br>\n",
    "- We have inserted these large embeddings into Pinecone\n",
    "<br>\n",
    "- When querying, each image is passed through the network and a large embedding is obtained\n",
    "<br>\n",
    "- We used this large embedding to find the most similar embeddings\n",
    "<br>\n",
    "- When similar images are found, we checked our vocabularies for ingredients and instructions\n",
    "<br>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "407a312e",
   "metadata": {},
   "source": [
    "Recipe generation model extracts image features e<sub>I</sub> with the image encoder, parametrized by θ<sub>I</sub> . Ingredients are predicted by θ<sub>L</sub>, and encoded into ingredient embeddings e<sub>L</sub> with θ<sub>e</sub>. The cooking instruction decoder, parametrized by θ<sub>R</sub> generates a recipe title and a sequence of cooking steps by attending to image embeddings e<sub>I</sub> , ingredient embeddings e<sub>L</sub> , and previously predicted words (r<sub>0</sub>, ..., r<sub>t−1</sub>)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e5bce01",
   "metadata": {},
   "source": [
    "<img src=\"images/facebook_model.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a27b0dfe",
   "metadata": {},
   "source": [
    "Besides showcasing Facebook's solution, we have also checked what happens if we use much smaller embeddings (~10 times smaller) obtained from the same convolutional network (ResNet101) that was pre-trained on the imagenet dataset, and the same vocabularies to search for ingredients and instructions. \n",
    "\n",
    "We do this by using Facebook's instructions and ingredients vocabularies with a modified image encoder. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75506ecc",
   "metadata": {},
   "source": [
    "Let's get started!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5737209",
   "metadata": {},
   "source": [
    "# Facebook's inversecooking model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05fffbd2",
   "metadata": {},
   "source": [
    "## Dependencies"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73e60f1a",
   "metadata": {},
   "source": [
    "Install and import relevant python packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d5f06f29",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install matplotlib\n",
    "!pip install torch\n",
    "!pip install numpy\n",
    "!pip install Pillow\n",
    "!pip install tensorflow\n",
    "!pip install gdown\n",
    "!pip install pandas\n",
    "!pip install pinecone-client==3.1.0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11c94498",
   "metadata": {},
   "source": [
    "We will now download already created ingredients and instructions vocabularies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2a89a69d",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# ingredients vocabulary\n",
    "!wget https://dl.fbaipublicfiles.com/inversecooking/ingr_vocab.pkl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f3153ce",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# instructions vocabulary\n",
    "!wget https://dl.fbaipublicfiles.com/inversecooking/instr_vocab.pkl"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a5e5e4f",
   "metadata": {},
   "source": [
    "Downloading Facebook's pretrained model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e92dc7f5",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!wget https://dl.fbaipublicfiles.com/inversecooking/modelbest.ckpt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "183a7772",
   "metadata": {},
   "source": [
    "We wil now clone Facebook's `inversecooking` repository. We are doing this because in the following steps we will be using some of the methods to access Facebook's model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d676307",
   "metadata": {},
   "outputs": [],
   "source": [
    "!git clone https://github.com/facebookresearch/inversecooking.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "6ec5cff2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import numpy as np\n",
    "import os\n",
    "from inversecooking.src.args import get_parser\n",
    "import pickle\n",
    "from torchvision import transforms\n",
    "from PIL import Image\n",
    "import time\n",
    "import requests\n",
    "from io import BytesIO\n",
    "import random\n",
    "from collections import Counter"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce6efb76",
   "metadata": {},
   "source": [
    "If you are planning on creating your own vectors, you should download a dataset from this link [Recipe1M](http://im2recipe.csail.mit.edu/dataset/download) (registration required). We have used the test set to demonstrate Pinecone's capabilities to handle large embeddings. Note that vector creation is a long process and should take around 10 hours for 10K vectors."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3cdea5c",
   "metadata": {},
   "source": [
    "On the other hand, if you would like to use already created vectors, in the next step we will download them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5bd4ef0e",
   "metadata": {},
   "outputs": [],
   "source": [
    "!gdown --id 1gE258b6bvIMiM7hk0T96R5Fpt5XNHkAf  # downloading ingredients\n",
    "!gdown --id 1tuTRqLrEU8cuUeQEx1TSFfFmJOfCLfYa  # downloading instructions\n",
    "!gdown --id 1ep62ohbWP6Ywk7WPf7-mlKSShuZpp-cT  # downloading vectors generated by Facebook's model\n",
    "!gdown --id 1PKzN2zdVBq5dlmDu2xACIUO3B6FAcS2_  # downloading vectors generated by a modified ResNet101"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "687853a9",
   "metadata": {},
   "source": [
    "Specify the directory to which you will copy your downloaded images. Feel free to leave images within their subdirectories. The following code will fetch them properly.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "7c792550",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dir = 'data'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "d04767b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "map_loc = None if torch.cuda.is_available() else 'cpu'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71493cd5",
   "metadata": {},
   "source": [
    "Load vocabularies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "f2b3bafd",
   "metadata": {},
   "outputs": [],
   "source": [
    "ingrs_vocab = pickle.load(open('ingr_vocab.pkl', 'rb'))\n",
    "vocab = pickle.load(open('instr_vocab.pkl', 'rb'))\n",
    "\n",
    "ingr_vocab_size = len(ingrs_vocab)\n",
    "instrs_vocab_size = len(vocab)\n",
    "output_dim = instrs_vocab_size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "7f9c2d7a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "23231 1488\n"
     ]
    }
   ],
   "source": [
    "print (instrs_vocab_size, ingr_vocab_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ab9405b",
   "metadata": {},
   "source": [
    "Load the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "310c94df",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys; sys.argv=['']; sys.path.insert(1, 'inversecooking/src')\n",
    "from inversecooking.src.model import get_model\n",
    "\n",
    "args = get_parser()\n",
    "args.maxseqlen = 15\n",
    "args.ingrs_only=False\n",
    "model = get_model(args, ingr_vocab_size, instrs_vocab_size)\n",
    "\n",
    "model_path = 'modelbest.ckpt'\n",
    "model.load_state_dict(torch.load(model_path, map_location=map_loc))\n",
    "model.to(device)\n",
    "model.eval()\n",
    "model.ingrs_only = False\n",
    "model.recipe_only = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "32f36b25",
   "metadata": {},
   "outputs": [],
   "source": [
    "transf_list_batch = []\n",
    "transf_list_batch.append(transforms.ToTensor())\n",
    "transf_list_batch.append(transforms.Normalize((0.485, 0.456, 0.406), \n",
    "                                              (0.229, 0.224, 0.225)))\n",
    "to_input_transf = transforms.Compose(transf_list_batch)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "304382ac",
   "metadata": {},
   "source": [
    "## Preparing data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8643a9d5",
   "metadata": {},
   "source": [
    "Now that we have loaded Facebook's pretrained model as well as ingredients and instructions vocabularies, we are ready to start creating embeddings."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccc44c42",
   "metadata": {},
   "source": [
    "You can choose to create image vectors or load prepared vectors. \n",
    "Setting `create_vectors` to `True` will, in the following steps, trigger vector creation from scratch. This is a long operation. Depending on the number of vectors you choose to create and whether you choose to use cpu or gpu, creating vectors can take around 10 hours.\n",
    "Setting `create_vectors` to `False` will load vectors that have already been created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "fcb83660",
   "metadata": {},
   "outputs": [],
   "source": [
    "create_vectors = False"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a203e09",
   "metadata": {},
   "source": [
    "Create a list of filepaths to all images in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "4dcd8d56",
   "metadata": {},
   "outputs": [],
   "source": [
    "if create_vectors:\n",
    "    show_anyways = False\n",
    "    image_folder = data_dir\n",
    "\n",
    "    demo_files = []\n",
    "    for path, subdirs, files in os.walk(image_folder):\n",
    "        for name in files:\n",
    "            if name.endswith('.jpg'):\n",
    "                demo_files.append(os.path.join(path, name))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e579aff",
   "metadata": {},
   "source": [
    "We will now prepare vectors for upload."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "4d331300",
   "metadata": {},
   "outputs": [],
   "source": [
    "if create_vectors:\n",
    "    items_to_upload = []\n",
    "    ingredients = {}\n",
    "    instructions = {}\n",
    "    \n",
    "    i=0\n",
    "    demo_files = demo_files[:10000]\n",
    "\n",
    "    for img_file in demo_files:\n",
    "        print(f'{i}/{len(demo_files)}', end='\\r')\n",
    "        i = i+1\n",
    "\n",
    "        image = Image.open(img_file).convert('RGB')\n",
    "\n",
    "        transf_list = []\n",
    "        transf_list.append(transforms.Resize(256))\n",
    "        transf_list.append(transforms.CenterCrop(224))\n",
    "        transform = transforms.Compose(transf_list)\n",
    "\n",
    "        image_transf = transform(image)\n",
    "        image_tensor = to_input_transf(image_transf).unsqueeze(0).to(device)\n",
    "\n",
    "        img_features = model.image_encoder(image_tensor)\n",
    "        features = model.image_encoder.linear(model.image_encoder.resnet(image_tensor))\n",
    "        \n",
    "        with torch.no_grad():\n",
    "            outputs = model.sample(image_tensor, \n",
    "                                   greedy=True, \n",
    "                                   temperature=1.0, \n",
    "                                   beam=-1, \n",
    "                                   true_ingrs=None)\n",
    "\n",
    "        ingr_ids = outputs['ingr_ids'].cpu().numpy()\n",
    "        recipe_ids = outputs['recipe_ids'].cpu().numpy()\n",
    "\n",
    "        items_to_upload.append((f'{img_file}', img_features.flatten().tolist()))\n",
    "\n",
    "        ingredients[img_file] = ingr_ids[0]\n",
    "        instructions[img_file] = recipe_ids[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27dcb916",
   "metadata": {},
   "source": [
    "In case you have chosen to create vectors from scratch, this step will save them so you don't have to recreate them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "72896a5c",
   "metadata": {},
   "outputs": [],
   "source": [
    "if create_vectors:\n",
    "    with open('vectors.pkl', 'wb') as f:\n",
    "        pickle.dump(items_to_upload, f)\n",
    "    with open('ingredients.pkl', 'wb') as f:\n",
    "        pickle.dump(ingredients, f)\n",
    "    with open('instructions.pkl', 'wb') as f:\n",
    "        pickle.dump(instructions, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24f27d89",
   "metadata": {},
   "source": [
    "Load already created vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "12b78e08",
   "metadata": {},
   "outputs": [],
   "source": [
    "if not create_vectors:\n",
    "    items_to_upload = pickle.load(open('vectors.pkl', 'rb'))\n",
    "    ingredients = pickle.load(open('ingredients.pkl', 'rb'))\n",
    "    instructions = pickle.load(open('instructions.pkl', 'rb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e448e798",
   "metadata": {},
   "source": [
    "Check vector size. This is the number of features in each vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "bfcc654e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25088"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(items_to_upload[0][1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdf97a4b",
   "metadata": {},
   "source": [
    "## Initializing the Index\n",
    "\n",
    "Now we need a place to store these embeddings and enable a efficient vector search through them all. To do that we use Pinecone, we can get a [free API key](https://app.pinecone.io/) and enter it below where we will initialize our connection to Pinecone and create a new index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b84fac69",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from pinecone import Pinecone\n",
    "\n",
    "# initialize connection to pinecone (get API key at app.pinecone.io)\n",
    "api_key = os.environ.get('PINECONE_API_KEY') or 'PINECONE_API_KEY'\n",
    "\n",
    "# configure client\n",
    "pc = Pinecone(api_key=api_key)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ead0af2",
   "metadata": {},
   "source": [
    "Now we setup our index specification, this allows us to define the cloud provider and region where we want to deploy our index. You can find a list of all [available providers and regions here](https://docs.pinecone.io/docs/projects)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e31ab865",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pinecone import ServerlessSpec\n",
    "\n",
    "cloud = os.environ.get('PINECONE_CLOUD') or 'aws'\n",
    "region = os.environ.get('PINECONE_REGION') or 'us-east-1'\n",
    "\n",
    "spec = ServerlessSpec(cloud=cloud, region=region)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abcf1675",
   "metadata": {},
   "source": [
    "Create the index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "1e01a3ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "index_name = 'recipe-search'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c80093bc",
   "metadata": {},
   "source": [
    "Now that we have our vector embeddings we can create our Pinecone service and upload the data to it.\n",
    "\n",
    "Below we create a Pinecone index. Here we use two arguments:\n",
    "\n",
    "metric=\"cosine\" means the vector search will use cosine as the measure of similarity.\n",
    "shards=3 means the service will run on three nodes. If you have more vectors, you should increase the number of shards. Refer to the documentation for guidelines on how to choose the number of shards."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d37eaa1b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "# check if index already exists (it shouldn't if this is first time)\n",
    "if index_name not in pc.list_indexes().names():\n",
    "    # if does not exist, create index\n",
    "    pc.create_index(\n",
    "        index_name,\n",
    "        dimension=len(items_to_upload[0][1]),\n",
    "        metric='cosine',\n",
    "        spec=spec\n",
    "    )\n",
    "    # wait for index to be initialized\n",
    "    while not pc.describe_index(index_name).status['ready']:\n",
    "        time.sleep(1)\n",
    "\n",
    "# connect to index\n",
    "index = pc.Index(index_name)\n",
    "# view index stats\n",
    "index.describe_index_stats()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07ee65bd",
   "metadata": {},
   "source": [
    "Upsert will upload, add, and index items into your remote vector similarity search service. If an item with that ID already exists it will be overwritten with the new provided value.\n",
    "\n",
    "Upsert is a long operation. It should be complete in 1-2 hours."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "015fb7fe",
   "metadata": {},
   "source": [
    "<img src='images/upsert.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "784435ce",
   "metadata": {},
   "source": [
    "## Upsert"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f7b950a",
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(0, len(items_to_upload), 100): \n",
    "    acks = index.upsert(vectors=items_to_upload[i:i+100])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e27918b",
   "metadata": {},
   "source": [
    "## Query"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a3af6bb",
   "metadata": {},
   "source": [
    "Now we will query for recipes for a number of food images from the internet.\n",
    "\n",
    "We are only looking for similar images in Pinecone's index and later check which ingredients and instructions are specified for each resulting image. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb5deb0a",
   "metadata": {},
   "source": [
    "<img src='images/query.png' />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "331127ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "# add step for ingredients and instructions\n",
    "demo_urls = [\n",
    "            'https://a7m3f5i5.rocketcdn.me/wp-content/uploads/2015/09/moms-spaghetti-sauce-recipe-a-healthy-slice-of-life-6-of-6-800x600.jpg',\n",
    "            'https://www.simplyrecipes.com/thmb/6azShcxiFtG0LlqXqz6eq3BM56A=/450x0/filters:no_upscale():max_bytes(150000):strip_icc()/__opt__aboutcom__coeus__resources__content_migration__simply_recipes__uploads__2007__03__spinach-horiz-640-2dcea783044340479666095cb40ab205.jpg',\n",
    "            'https://sugargeekshow.com/wp-content/uploads/2020/10/baked_donut_recipe_featured.jpg',\n",
    "            'https://www.moulinex-me.com/medias/?context=bWFzdGVyfHJvb3R8MTQzNTExfGltYWdlL2pwZWd8aDM2L2g1Mi8xMzA5NzI3MzI2MjExMC5qcGd8N2MxZDhmNmI5ZTgzZDZlZWQyZGQ4YjFlZjUyNDlkMTExYjdkZDdlZmFkY2I0N2NmNjljOGViNmExZjIyMDU4Yw',\n",
    "            'https://hips.hearstapps.com/hmg-prod/images/delish-how-to-make-a-smoothie-horizontal-1542310071.png',\n",
    "            'https://www.simplyrecipes.com/thmb/45-YxKaW3K8CYhYzfxnYUDhz63Q=/1800x1012/smart/filters:no_upscale()/__opt__aboutcom__coeus__resources__content_migration__simply_recipes__uploads__2007__01__roast-chicken-carrots-sally-horiz-a-1800-51888dd5b8c44b6e9e656092f4f96f12.jpg',\n",
    "            'https://www.jessicagavin.com/wp-content/uploads/2018/02/greek-salad-2-1200.jpg',\n",
    "            'https://www.twopeasandtheirpod.com/wp-content/uploads/2021/06/Greek-Quinoa-Salad4637.jpg',\n",
    "            'https://www.onceuponachef.com/images/2020/02/Double-Chocolate-Muffins.jpg',\n",
    "            'https://hips.hearstapps.com/del.h-cdn.co/assets/17/31/1501791674-delish-chicken-curry-horizontal.jpg?crop=1.00xw:0.750xh;0,0.159xh&resize=640:*'\n",
    "            ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "324ad1fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "from inversecooking.src.utils.output_utils import prepare_output\n",
    "from PIL import Image\n",
    "\n",
    "\n",
    "titles_display = []\n",
    "ingredients_display = []\n",
    "instructions_display = []\n",
    "images_display = []\n",
    "\n",
    "\n",
    "thresh_score = 0.4\n",
    "\n",
    "for img_file in demo_urls:\n",
    "    response = requests.get(img_file)\n",
    "    print(\"got response\")\n",
    "    image = Image.open(BytesIO(response.content))\n",
    "\n",
    "    transf_list = []\n",
    "    transf_list.append(transforms.Resize(256))\n",
    "    transf_list.append(transforms.CenterCrop(224))\n",
    "    transform = transforms.Compose(transf_list)\n",
    "    print(\"got transform\")\n",
    "\n",
    "    image_transf = transform(image)\n",
    "    image_tensor = to_input_transf(image_transf).unsqueeze(0).to(device)\n",
    "    print(\"got tensor\")\n",
    "\n",
    "    img_features = model.image_encoder(image_tensor)\n",
    "    sample_embedding = img_features.flatten().tolist()\n",
    "    print(\"Got embedding\")\n",
    "    \n",
    "    query_results = index.query(vector=sample_embedding, top_k=2, disable_progress_bar=True)\n",
    "    print(\"got results\")\n",
    "    \n",
    "    for image_path, score in zip(query_results[0].ids, query_results[0].scores):\n",
    "        if score > thresh_score:\n",
    "            image = Image.open(image_path).convert('RGB')\n",
    "\n",
    "            transf_list = []\n",
    "            transf_list.append(transforms.Resize(256))\n",
    "            transf_list.append(transforms.CenterCrop(224))\n",
    "            transform = transforms.Compose(transf_list)\n",
    "\n",
    "            image_transf = transform(image)\n",
    "\n",
    "            images_display.append(image_path)\n",
    "        \n",
    "            outs, valid = prepare_output(instructions[img_file],\n",
    "                                         ingredients[img_file], \n",
    "                                         ingrs_vocab, \n",
    "                                         vocab)\n",
    "\n",
    "            if not valid:\n",
    "                print (\"Not a valid recipe!\")\n",
    "                print (\"Reason: \", valid['reason'])\n",
    "                continue\n",
    "            else:\n",
    "                titles_display.append(outs['title'])\n",
    "                ingredients_display.append(', '.join(outs['ingrs']))\n",
    "                instructions_display.append('-'+'\\n-'.join(outs['recipe']))\n",
    "\n",
    "        else:\n",
    "            print('No recipes found.')\n",
    "            break"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91bbba61",
   "metadata": {},
   "source": [
    "## Display results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "59c8814c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from IPython.display import Image, display\n",
    "from IPython.core.display import HTML \n",
    "\n",
    "\n",
    "def path_to_image_html(path):\n",
    "     return '<img src=\"'+ path + '\" width=\"200\"/>'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "84aabd6b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<th style=\"text-align:left\"><td style=\"vertical-align:top\"><h2>Results</h2><table style=\"display:inline;\" width=\"100%\" border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th text-align=\"left\"></th>\n",
       "      <th text-align=\"left\">query_image</th>\n",
       "      <th text-align=\"left\">similar recipe</th>\n",
       "      <th text-align=\"left\">recipe_image</th>\n",
       "      <th text-align=\"left\">ingredients</th>\n",
       "      <th text-align=\"left\">instructions</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">0</th>\n",
       "      <td text-align=\"left\"><img src=\"https://a7m3f5i5.rocketcdn.me/wp-content/uploads/2015/09/moms-spaghetti-sauce-recipe-a-healthy-slice-of-life-6-of-6-800x600.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Spaghetti sauce</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/2/b/a/92ba384999.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">onion, pepper, tomato, clove, oil, salt, beef, oregano, basil, parsley</td>\n",
       "      <td text-align=\"left\">-In a large skillet, heat oil over medium heat.<br>-Add onion and garlic and saute until tender.<br>-Add ground beef and cook until browned.<br>-Add tomatoes, tomato paste, parsley, basil, oregano, salt and pepper.<br>-Simmer for 30 minutes.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">1</th>\n",
       "      <td text-align=\"left\"><img src=\"https://a7m3f5i5.rocketcdn.me/wp-content/uploads/2015/09/moms-spaghetti-sauce-recipe-a-healthy-slice-of-life-6-of-6-800x600.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Spaghetti with tomato sauce</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/7/3/6/97367e053a.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">oil, pepper, tomato, clove, spaghetti, parsley, salt, cheese, basil, wine, juice, onion</td>\n",
       "      <td text-align=\"left\">-In a large pot of boiling salted water, cook the spaghetti until al dente.<br>-Drain and return to the pot.<br>-Meanwhile, in a large, deep skillet, heat the olive oil.<br>-Add the onion and cook over moderate heat, stirring occasionally, until softened, about 5 minutes.<br>-Add the garlic and cook, stirring, until fragrant, about 1 minute.<br>-Add the tomatoes, wine, lemon juice, basil, salt and pepper and bring to a boil.<br>-Simmer over moderate heat, stirring occasionally, until the sauce is thickened, about 20 minutes.<br>-Add the spaghetti and toss well.<br>-Add the parmesan and toss well.<br>-Serve right away.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">2</th>\n",
       "      <td text-align=\"left\"><img src=\"https://www.simplyrecipes.com/thmb/6azShcxiFtG0LlqXqz6eq3BM56A=/450x0/filters:no_upscale():max_bytes(150000):strip_icc()/__opt__aboutcom__coeus__resources__content_migration__simply_recipes__uploads__2007__03__spinach-horiz-640-2dcea783044340479666095cb40ab205.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Garlicky sauteed kale</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/b/f/f/9bff4dc9b8.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">oil, kale, pepper, salt, clove, spinach</td>\n",
       "      <td text-align=\"left\">-Heat olive oil in a large skillet over medium heat.<br>-Add garlic and cook until fragrant, about 30 seconds.<br>-Add kale and cook until wilted, about 2 minutes.<br>-Season with salt and pepper.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">3</th>\n",
       "      <td text-align=\"left\"><img src=\"https://www.simplyrecipes.com/thmb/6azShcxiFtG0LlqXqz6eq3BM56A=/450x0/filters:no_upscale():max_bytes(150000):strip_icc()/__opt__aboutcom__coeus__resources__content_migration__simply_recipes__uploads__2007__03__spinach-horiz-640-2dcea783044340479666095cb40ab205.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Sauteed spinach with garlic</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/0/4/2/9042144b09.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">oil, spinach, clove, salt, soy_sauce, pepper</td>\n",
       "      <td text-align=\"left\">-Heat oil in a large skillet over medium-high heat.<br>-Add garlic; saute 30 seconds.<br>-Add spinach; saute 1 minute.<br>-Add soy sauce, salt, and pepper; toss to coat.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">4</th>\n",
       "      <td text-align=\"left\"><img src=\"https://sugargeekshow.com/wp-content/uploads/2020/10/baked_donut_recipe_featured.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Vanilla cupcakes</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/b/0/c/9b0c937484.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">sugar, flour, butter, salt, egg, baking_powder, milk, extract, vanilla</td>\n",
       "      <td text-align=\"left\">-Preheat oven to 350 degrees.<br>-Line muffin tins with paper liners.<br>-In a large bowl, cream butter and sugar until light and fluffy.<br>-Add eggs, one at a time, beating well after each addition.<br>-Add vanilla and milk.<br>-In a separate bowl, combine flour, baking powder and salt.<br>-Add dry ingredients to creamed mixture alternately with milk, beginning and ending with dry ingredients.<br>-Fill muffin cups 2/3 full.<br>-Bake for 20-25 minutes or until a toothpick inserted in the center comes out clean.<br>-Cool in pan for 5 minutes before removing to a wire rack to cool completely.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">5</th>\n",
       "      <td text-align=\"left\"><img src=\"https://sugargeekshow.com/wp-content/uploads/2020/10/baked_donut_recipe_featured.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Sugar cookies</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/c/4/b/9c4b70b316.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">sugar, flour, egg, salt, butter, baking_powder, vanilla, extract, baking_soda</td>\n",
       "      <td text-align=\"left\">-Preheat oven to 350 degrees.<br>-Cream butter and sugar.<br>-Add eggs and vanilla.<br>-Mix well.<br>-Add dry ingredients and mix well.<br>-Roll dough into 1 inch balls and place on ungreased cookie sheet.<br>-Flatten with bottom of glass dipped in sugar.<br>-Bake for 8-10 minutes.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">6</th>\n",
       "      <td text-align=\"left\"><img src=\"https://www.moulinex-me.com/medias/?context=bWFzdGVyfHJvb3R8MTQzNTExfGltYWdlL2pwZWd8aDM2L2g1Mi8xMzA5NzI3MzI2MjExMC5qcGd8N2MxZDhmNmI5ZTgzZDZlZWQyZGQ4YjFlZjUyNDlkMTExYjdkZDdlZmFkY2I0N2NmNjljOGViNmExZjIyMDU4Yw\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Pizza dough</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/9/2/f/992f2b1a75.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">salt, flour, water, yeast, cheese, tomato, oil</td>\n",
       "      <td text-align=\"left\">-Mix yeast and warm water in a large bowl.<br>-Let stand for 5 minutes.<br>-Add flour and salt.<br>-Mix well.<br>-Knead on a floured surface for 5 minutes.<br>-Place in a greased bowl and cover with a towel.<br>-Let rise in a warm place for 1 hour.<br>-Punch down and divide into 12 pieces.<br>-Roll each piece into a ball.<br>-Place on a greased baking sheet.<br>-Cover and let rise for 1 hour.<br>-Preheat oven to 450 degrees.<br>-Brush with olive oil.<br>-Bake for 10 minutes.<br>-Remove from oven and sprinkle with cheese.<br>-Bake for another 5 minutes.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">7</th>\n",
       "      <td text-align=\"left\"><img src=\"https://www.moulinex-me.com/medias/?context=bWFzdGVyfHJvb3R8MTQzNTExfGltYWdlL2pwZWd8aDM2L2g1Mi8xMzA5NzI3MzI2MjExMC5qcGd8N2MxZDhmNmI5ZTgzZDZlZWQyZGQ4YjFlZjUyNDlkMTExYjdkZDdlZmFkY2I0N2NmNjljOGViNmExZjIyMDU4Yw\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Pizza dough</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/2/f/4/92f46418ed.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">salt, flour, water, yeast, sugar</td>\n",
       "      <td text-align=\"left\">-Mix all ingredients together.<br>-Knead for 5 minutes.<br>-Let rise for 30 minutes.<br>-Roll out and cut into desired shapes.<br>-Bake at 400 for 15 minutes.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">8</th>\n",
       "      <td text-align=\"left\"><img src=\"https://hips.hearstapps.com/hmg-prod/images/delish-how-to-make-a-smoothie-horizontal-1542310071.png\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Banana smoothie</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/f/c/1/9fc1fa1c45.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">banana, milk, juice, ice</td>\n",
       "      <td text-align=\"left\">-Place all ingredients in a blender and blend until smooth.<br>-Pour into 2 tall glasses and enjoy !</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">9</th>\n",
       "      <td text-align=\"left\"><img src=\"https://hips.hearstapps.com/hmg-prod/images/delish-how-to-make-a-smoothie-horizontal-1542310071.png\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Strawberry smoothie</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/3/a/2/93a2286885.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">strawberries, yogurt, ice, juice</td>\n",
       "      <td text-align=\"left\">-Place all ingredients in blender.<br>-Blend until smooth.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">10</th>\n",
       "      <td text-align=\"left\"><img src=\"https://www.simplyrecipes.com/thmb/45-YxKaW3K8CYhYzfxnYUDhz63Q=/1800x1012/smart/filters:no_upscale()/__opt__aboutcom__coeus__resources__content_migration__simply_recipes__uploads__2007__01__roast-chicken-carrots-sally-horiz-a-1800-51888dd5b8c44b6e9e656092f4f96f12.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Roasted chicken</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/5/1/d/951da7fd7d.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">pepper, chicken, salt, onion, garlic, thyme, clove</td>\n",
       "      <td text-align=\"left\">-Preheat oven to 400 degrees f (200 degrees c).<br>-Place chicken in a roasting pan.<br>-Sprinkle with garlic powder, onion powder, thyme, salt, and pepper.<br>-Pour the onion soup mix over the chicken.<br>-Bake in the preheated oven for 1 hour, or until the chicken is no longer pink inside and the juices run clear.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">11</th>\n",
       "      <td text-align=\"left\"><img src=\"https://www.simplyrecipes.com/thmb/45-YxKaW3K8CYhYzfxnYUDhz63Q=/1800x1012/smart/filters:no_upscale()/__opt__aboutcom__coeus__resources__content_migration__simply_recipes__uploads__2007__01__roast-chicken-carrots-sally-horiz-a-1800-51888dd5b8c44b6e9e656092f4f96f12.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Chicken breast with mushrooms</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/0/9/8/b/098b7b0ea6.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">pepper, chicken, mushroom, onion, oil, butter, salt</td>\n",
       "      <td text-align=\"left\">-Preheat oven to 350 degrees f (175 degrees c).<br>-Heat olive oil in a large skillet over medium heat.<br>-Season chicken breasts with salt and pepper.<br>-Cook chicken breasts in hot oil until browned on both sides, about 5 minutes per side.<br>-Transfer chicken to a 9x13-inch baking dish.<br>-Melt butter in the same skillet over medium heat; cook and stir mushrooms and onion in the melted butter until mushrooms are tender, about 5 minutes.<br>-Pour mushroom mixture over chicken breasts.<br>-Bake in the preheated oven until chicken is no longer pink in the center and the juices run clear, about 1 hour.<br>-An instant-read thermometer inserted into the center should read at least 165 degrees f (74 degrees c).</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">12</th>\n",
       "      <td text-align=\"left\"><img src=\"https://www.jessicagavin.com/wp-content/uploads/2018/02/greek-salad-2-1200.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Greek salad</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/3/c/3/93c36ab080.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">tomato, cheese, cucumber, olive, pepper, oil, onion, vinegar, salt, oregano, lettuce</td>\n",
       "      <td text-align=\"left\">-Combine first 6 ingredients in a large bowl.<br>-Add dressing; toss to coat.<br>-Sprinkle with cheese.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">13</th>\n",
       "      <td text-align=\"left\"><img src=\"https://www.jessicagavin.com/wp-content/uploads/2018/02/greek-salad-2-1200.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Greek village salad</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/0/8/0/9080cc6143.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">onion, olive, tomato, cucumber, cheese, oil, pepper, salt, vinegar</td>\n",
       "      <td text-align=\"left\">-Combine vegetable style=\"display:inline;\" width=\"100%\"s in a large bowl.<br>-Add cheese and olives.<br>-Whisk together the remaining ingredients and pour over salad.<br>-Toss to coat.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">14</th>\n",
       "      <td text-align=\"left\"><img src=\"https://www.twopeasandtheirpod.com/wp-content/uploads/2021/06/Greek-Quinoa-Salad4637.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Greek salad</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/f/3/5/9f3549c916.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">tomato, cheese, oil, olive, pepper, onion, cucumber, vinegar, salt, oregano, basil</td>\n",
       "      <td text-align=\"left\">-Combine all ingredients in a large bowl.<br>-Toss well.<br>-Serve immediately.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">15</th>\n",
       "      <td text-align=\"left\"><img src=\"https://www.twopeasandtheirpod.com/wp-content/uploads/2021/06/Greek-Quinoa-Salad4637.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Quinoa salad</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/1/4/2/91428dbbfa.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">tomato, oil, pepper, quinoa, salt, couscous, juice, onion, water</td>\n",
       "      <td text-align=\"left\">-Bring water to a boil in a saucepan.<br>-Stir in quinoa, reduce heat to medium-low, and simmer until quinoa is tender and water has been absorbed, about 15 minutes.<br>-Remove from heat and let stand until cool enough to handle, about 10 minutes.<br>-Fluff quinoa with a fork.<br>-Mix quinoa, tomatoes, green onions, and olive oil together in a large bowl.<br>-Season with salt and black pepper.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">16</th>\n",
       "      <td text-align=\"left\"><img src=\"https://www.onceuponachef.com/images/2020/02/Double-Chocolate-Muffins.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Chocolate cupcakes</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/b/2/9/9b2977db32.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">sugar, flour, egg, cocoa, salt, baking_powder, milk, extract, butter, baking_soda, oil</td>\n",
       "      <td text-align=\"left\">-Preheat oven to 350 degrees f (175 degrees c).<br>-Grease and flour a muffin pan.<br>-In a large bowl, cream together the butter and sugar until light and fluffy.<br>-Beat in the eggs one at a time, then stir in the vanilla.<br>-Combine the flour, cocoa, baking powder, baking soda and salt; stir into the creamed mixture alternately with the milk.<br>-Fill the prepared muffin cups 2/3 full.<br>-Bake in the preheated oven for 20 to 25 minutes, or until a toothpick inserted into the center of a cupcake comes out clean.<br>-Allow to cool in the pan for 10 minutes before removing to a wire rack to cool completely.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">17</th>\n",
       "      <td text-align=\"left\"><img src=\"https://www.onceuponachef.com/images/2020/02/Double-Chocolate-Muffins.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Chocolate chip cookies</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/9/9/9/99990510b0.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">sugar, flour, egg, chips, salt, cocoa, extract, butter, baking_soda, chocolate, baking_powder</td>\n",
       "      <td text-align=\"left\">-Preheat oven to 350 degrees f (175 degrees c).<br>-Grease cookie sheets.<br>-In a large bowl, cream together the butter and sugar until smooth.<br>-Beat in the eggs one at a time, then stir in the vanilla.<br>-Combine the flour, cocoa, baking powder, baking soda and salt; stir into the creamed mixture.<br>-Finally, stir in the chocolate chips.<br>-Drop by rounded spoonfuls onto the prepared cookie sheets.<br>-Bake for 8 to 10 minutes in the preheated oven.<br>-Allow cookies to cool on baking sheet for 5 minutes before removing to a wire rack to cool completely.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">18</th>\n",
       "      <td text-align=\"left\"><img src=\"https://hips.hearstapps.com/del.h-cdn.co/assets/17/31/1501791674-delish-chicken-curry-horizontal.jpg?crop=1.00xw:0.750xh;0,0.159xh&resize=640:*\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Chana masala</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/0/2/b/902b10ba86.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">onion, oil, potato, salt, turmeric, clove, tomato, pepper, curry, coriander, chickpeas, ginger, cumin, masala, chili, paprika</td>\n",
       "      <td text-align=\"left\">-Heat oil in a large pan and add the onion and garlic.<br>-Cook until soft.<br>-Add the spices and cook for a minute.<br>-Add the tomatoes and cook for a few minutes.<br>-Add the potatoes and cook for a few minutes.<br>-Add the chickpeas and cook for a few minutes.<br>-Serve with rice.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">19</th>\n",
       "      <td text-align=\"left\"><img src=\"https://hips.hearstapps.com/del.h-cdn.co/assets/17/31/1501791674-delish-chicken-curry-horizontal.jpg?crop=1.00xw:0.750xh;0,0.159xh&resize=640:*\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Chicken curry</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/a/8/6/9a8628fe6c.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">onion, oil, salt, clove, chicken, turmeric, ginger, coriander, curry, masala, seeds, milk, pepper, tomato, chili</td>\n",
       "      <td text-align=\"left\">-Heat oil in a large pan and add the mustard seeds.<br>-When they start to pop, add the onions and garlic.<br>-Cook until they start to soften.<br>-Add the ginger, chilli powder, turmeric, coriander powder, garam masala, salt and pepper.<br>-Cook for a few minutes.<br>-Add the tomatoes and cook for a few minutes.<br>-Add the chicken and cook for a few minutes.<br>-Add the coconut milk and bring to the boil.<br>-Cover and simmer for about 20 minutes.<br>-Serve with rice.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table style=\"display:inline;\" width=\"100%\"></td></th>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import display_html\n",
    "from itertools import chain,cycle\n",
    "import pandas as pd\n",
    "\n",
    "def display_table(*args,titles=cycle([''])):\n",
    "    html_str=''\n",
    "    for df,title in zip(args, chain(titles,cycle(['</br>'])) ):\n",
    "        html_str+='<th style=\"text-align:left\"><td style=\"vertical-align:top\">'\n",
    "        html_str+=f'<h2>{title}</h2>'\n",
    "        html_str+=df.to_html(escape=False, \n",
    "                             formatters=dict(query_image=path_to_image_html, \n",
    "                                             recipe_image=path_to_image_html)\n",
    "                            ).replace('table','table style=\"display:inline;\" width=\"100%\"')\n",
    "        html_str+='</td></th>'\n",
    "    html_str = html_str.replace('<td>', '<td text-align=\"left\">')\n",
    "    html_str = html_str.replace('<th>', '<th text-align=\"left\">')\n",
    "    display_html(html_str,raw=True)\n",
    "\n",
    "    \n",
    "\n",
    "\n",
    "# Results\n",
    "instructions_display = [instruction.replace('\\n', '<br>') for instruction in instructions_display]\n",
    "\n",
    "df_r = pd.DataFrame({'query_image': [entry for entry in demo_urls for _ in range(2)],\n",
    "                     'similar recipe': titles_display, \n",
    "                     'recipe_image': images_display,\n",
    "                     'ingredients': ingredients_display, \n",
    "                     'instructions': instructions_display})\n",
    "\n",
    "# Show results\n",
    "display_table(df_r, titles=['Results'])\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00c44a1a",
   "metadata": {},
   "source": [
    "# Pretrained ResNet101"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ede6682",
   "metadata": {},
   "source": [
    "In this section we will be showing how you can use a pre-trained network (trained on imagenet dataset), create much smaller embeddings and take advantage of Facebook's ingredients and instructions vocabularies.\n",
    "\n",
    "You will see that results are comparable, even though we are working with much smaller embeddings."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e165a77c",
   "metadata": {},
   "source": [
    "As before, we will start by importing necessary packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "c5bf427d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.applications import ResNet101\n",
    "from tensorflow.keras.preprocessing import image\n",
    "from tensorflow.keras.applications.resnet import preprocess_input, decode_predictions\n",
    "from tensorflow import keras"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "dca11834",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "requests.packages.urllib3.disable_warnings()\n",
    "import ssl\n",
    "\n",
    "try:\n",
    "    _create_unverified_https_context = ssl._create_unverified_context\n",
    "except AttributeError:\n",
    "    pass\n",
    "else:\n",
    "    ssl._create_default_https_context = _create_unverified_https_context"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2726e30",
   "metadata": {},
   "source": [
    "We will now create a custom model by taking the ResNet model and removing the last layer. This network is pre-trained on imagenet dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a147b264",
   "metadata": {},
   "outputs": [],
   "source": [
    "resnet_model = ResNet101(include_top=True, \n",
    "                         weights='imagenet', \n",
    "                         input_tensor=None, \n",
    "                         input_shape=None, \n",
    "                         pooling=None, \n",
    "                         classes=1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "80e8ec9e",
   "metadata": {},
   "outputs": [],
   "source": [
    "cust_model = keras.Model(resnet_model.input, resnet_model.layers[-2].output)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b903df5f",
   "metadata": {},
   "source": [
    "We will now create embeddings using this slightly modified network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "88ab3c3e",
   "metadata": {},
   "outputs": [],
   "source": [
    "if create_vectors:\n",
    "    items_to_upload_cust = []\n",
    "\n",
    "    i=1\n",
    "    for img_file in demo_files:\n",
    "        print(f'{i}/{len(demo_files)}', end='\\r')\n",
    "        i = i+1\n",
    "\n",
    "        img = image.load_img(img_file, target_size=(224, 224))\n",
    "        x = image.img_to_array(img)\n",
    "        x = np.expand_dims(x, axis=0)\n",
    "        x = preprocess_input(x)\n",
    "\n",
    "        features = cust_model.predict(x)\n",
    "\n",
    "        items_to_upload_cust.append((f'{img_file}', features[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d12a173",
   "metadata": {},
   "source": [
    "To avoid waiting for vectors to be created, we are going to save them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "a5b85411",
   "metadata": {},
   "outputs": [],
   "source": [
    "if create_vectors:\n",
    "    with open('vectors_cust.pkl', 'wb') as f:\n",
    "        pickle.dump(items_to_upload_cust, f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "844cff7d",
   "metadata": {},
   "outputs": [],
   "source": [
    "if not create_vectors:\n",
    "    items_to_upload_cust = pickle.load(open('vectors_cust.pkl', 'rb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "894b6a32",
   "metadata": {},
   "source": [
    "We will now create a new index and upsert vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "92a7716a",
   "metadata": {},
   "outputs": [],
   "source": [
    "index_name_cust = 'recipe-search-custom'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "237e7086",
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "# check if index already exists (it shouldn't if this is first time)\n",
    "if index_name_cust not in pc.list_indexes().names():\n",
    "    # if does not exist, create index\n",
    "    pc.create_index(\n",
    "        index_name_cust,\n",
    "        dimension=1000,\n",
    "        metric='cosine',\n",
    "        spec=spec\n",
    "    )\n",
    "    # wait for index to be initialized\n",
    "    while not pc.describe_index(index_name_cust).status['ready']:\n",
    "        time.sleep(1)\n",
    "\n",
    "# connect to index\n",
    "index_cust = pc.Index(index_name_cust)\n",
    "# view index stats\n",
    "index_cust.describe_index_stats()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5f243e4",
   "metadata": {},
   "source": [
    "## Upsert"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "8ffe2f4a",
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(0, len(items_to_upload_cust), 1000): \n",
    "    acks = index_cust.upsert(vectors=items_to_upload_cust[i:i+1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1fc65ab2",
   "metadata": {},
   "source": [
    "## Query"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbabce7c",
   "metadata": {},
   "source": [
    "Finally, we will use the same test images to check the performance of our custom model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "1d3dc14c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from inversecooking.src.utils.output_utils import prepare_output\n",
    "from tensorflow.keras.preprocessing import image\n",
    "import urllib\n",
    "from PIL import Image\n",
    "\n",
    "\n",
    "titles_display_cust = []\n",
    "ingredients_display_cust = []\n",
    "instructions_display_cust = []\n",
    "images_display_cust = []\n",
    "\n",
    "thresh_score = 0.4\n",
    "\n",
    "for img_file in demo_urls:\n",
    "    response = requests.get(img_file)\n",
    "    img = Image.open(BytesIO(response.content))\n",
    "    \n",
    "    transf_list = []\n",
    "    transf_list.append(transforms.Resize(256))\n",
    "    transf_list.append(transforms.CenterCrop(224))\n",
    "    transform = transforms.Compose(transf_list)\n",
    "\n",
    "    image_transf = transform(img)\n",
    "\n",
    "    x = image.img_to_array(image_transf)\n",
    "    x = np.expand_dims(x, axis=0)\n",
    "    x = preprocess_input(x)\n",
    "\n",
    "    features = cust_model.predict(x)[0]\n",
    "    \n",
    "    query_results = index_cust.query(vector=features, top_k=2, disable_progress_bar=True)\n",
    "    \n",
    "    for img_file, score in zip(query_results[0].ids, query_results[0].scores):\n",
    "        if score > thresh_score:\n",
    "          \n",
    "            images_display_cust.append(img_file)\n",
    "\n",
    "            outs, valid = prepare_output(instructions[img_file],\n",
    "                                         ingredients[img_file], \n",
    "                                         ingrs_vocab, \n",
    "                                         vocab)\n",
    "\n",
    "            if not valid:\n",
    "                print (\"Not a valid recipe!\")\n",
    "                print (\"Reason: \", valid['reason'])\n",
    "                continue\n",
    "            else:\n",
    "                titles_display_cust.append(outs['title'])\n",
    "                ingredients_display_cust.append(', '.join(outs['ingrs']))\n",
    "                instructions_display_cust.append('-'+'\\n-'.join(outs['recipe']))\n",
    "\n",
    "\n",
    "        else:\n",
    "            print('No recipes found.')\n",
    "            break"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbbac8a1",
   "metadata": {},
   "source": [
    "## Display results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "5ec13913",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<th style=\"text-align:left\"><td style=\"vertical-align:top\"><h2>Results</h2><table style=\"display:inline;\" width=\"100%\" border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th text-align=\"left\"></th>\n",
       "      <th text-align=\"left\">query_image</th>\n",
       "      <th text-align=\"left\">similar recipe</th>\n",
       "      <th text-align=\"left\">recipe_image</th>\n",
       "      <th text-align=\"left\">ingredients</th>\n",
       "      <th text-align=\"left\">instructions</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">0</th>\n",
       "      <td text-align=\"left\"><img src=\"https://a7m3f5i5.rocketcdn.me/wp-content/uploads/2015/09/moms-spaghetti-sauce-recipe-a-healthy-slice-of-life-6-of-6-800x600.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Pasta with tomato sauce</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/8/1/e/981eedb119.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">oil, pepper, tomato, clove, onion, salt, pasta, parsley, basil</td>\n",
       "      <td text-align=\"left\">-Heat the olive oil in a large skillet over medium heat.<br>-Add the onion and garlic and cook until the onion is translucent, about 5 minutes.<br>-Add the tomatoes, tomato paste, salt, pepper, and basil.<br>-Bring to a boil, then reduce the heat to low and simmer for about 20 minutes.<br>-Meanwhile, cook the pasta according to the package directions.<br>-Drain the pasta and add it to the tomato sauce.<br>-Toss to coat.<br>-Serve immediately.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">1</th>\n",
       "      <td text-align=\"left\"><img src=\"https://a7m3f5i5.rocketcdn.me/wp-content/uploads/2015/09/moms-spaghetti-sauce-recipe-a-healthy-slice-of-life-6-of-6-800x600.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Parmesan omelet</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/7/3/9/9739a6a9dd.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">pepper, oil, egg, butter, cheese, tomato, salt</td>\n",
       "      <td text-align=\"left\">-In a small bowl, beat eggs with salt and pepper.<br>-In a large skillet, heat oil over medium heat.<br>-Add eggs and cook, stirring constantly, until eggs are set.<br>-Remove from heat.<br>-Add cheese and tomatoes to eggs.<br>-Fold in omelet and serve.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">2</th>\n",
       "      <td text-align=\"left\"><img src=\"https://www.simplyrecipes.com/thmb/6azShcxiFtG0LlqXqz6eq3BM56A=/450x0/filters:no_upscale():max_bytes(150000):strip_icc()/__opt__aboutcom__coeus__resources__content_migration__simply_recipes__uploads__2007__03__spinach-horiz-640-2dcea783044340479666095cb40ab205.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Komatsuna namul</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/b/d/6/9bd6794d87.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">seeds, soy_sauce, salt, sugar, oil</td>\n",
       "      <td text-align=\"left\">-Parboil the komatsuna in salted water, then squeeze out excess water.<br>-Cut into 5 cm lengths.<br>-Combine the ingredients in a bowl, and mix well.<br>-Add the komatsuna and toss.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">3</th>\n",
       "      <td text-align=\"left\"><img src=\"https://www.simplyrecipes.com/thmb/6azShcxiFtG0LlqXqz6eq3BM56A=/450x0/filters:no_upscale():max_bytes(150000):strip_icc()/__opt__aboutcom__coeus__resources__content_migration__simply_recipes__uploads__2007__03__spinach-horiz-640-2dcea783044340479666095cb40ab205.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Mongolian beef</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/1/4/6/91469e4d34.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">soy_sauce, oil, sugar, steak, ginger, clove, onion, pepper</td>\n",
       "      <td text-align=\"left\">-In a small bowl, combine soy sauce, brown sugar, sesame oil, garlic, ginger and red pepper flakes.<br>-Place steak in a large resealable plastic bag.<br>-Pour soy sauce mixture over steak, seal bag, and turn to coat.<br>-Marinate in the refrigerator for at least 1 hour, turning occasionally.<br>-Preheat an outdoor grill for medium-high heat and lightly oil grate.<br>-Remove steak from marinade, and discard marinade.<br>-Grill steak on preheated grill until desired doneness, about 5 minutes per side for medium.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">4</th>\n",
       "      <td text-align=\"left\"><img src=\"https://sugargeekshow.com/wp-content/uploads/2020/10/baked_donut_recipe_featured.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Vanilla cupcakes</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/b/0/c/9b0c937484.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">sugar, flour, butter, salt, egg, baking_powder, milk, extract, vanilla</td>\n",
       "      <td text-align=\"left\">-Preheat oven to 350 degrees.<br>-Line muffin tins with paper liners.<br>-In a large bowl, cream butter and sugar until light and fluffy.<br>-Add eggs, one at a time, beating well after each addition.<br>-Add vanilla and milk.<br>-In a separate bowl, combine flour, baking powder and salt.<br>-Add dry ingredients to creamed mixture alternately with milk, beginning and ending with dry ingredients.<br>-Fill muffin cups 2/3 full.<br>-Bake for 20-25 minutes or until a toothpick inserted in the center comes out clean.<br>-Cool in pan for 5 minutes before removing to a wire rack to cool completely.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">5</th>\n",
       "      <td text-align=\"left\"><img src=\"https://sugargeekshow.com/wp-content/uploads/2020/10/baked_donut_recipe_featured.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Candy cane sugar</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/e/5/2/9e526f76da.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">sugar, food_coloring, water, candies</td>\n",
       "      <td text-align=\"left\">-In a large bowl, combine the confectioners' sugar, water and peppermint candies.<br>-Mix until well blended.<br>-Tint with food coloring if desired.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">6</th>\n",
       "      <td text-align=\"left\"><img src=\"https://www.moulinex-me.com/medias/?context=bWFzdGVyfHJvb3R8MTQzNTExfGltYWdlL2pwZWd8aDM2L2g1Mi8xMzA5NzI3MzI2MjExMC5qcGd8N2MxZDhmNmI5ZTgzZDZlZWQyZGQ4YjFlZjUyNDlkMTExYjdkZDdlZmFkY2I0N2NmNjljOGViNmExZjIyMDU4Yw\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Pizza dough</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/0/4/3/90435bffb6.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">cheese, tomato, pepper, pizza_dough, salt, yeast, oil, flour, water</td>\n",
       "      <td text-align=\"left\">-In a large bowl, dissolve yeast in warm water.<br>-Let stand until creamy, about 10 minutes.<br>-Stir in flour, salt, and oil.<br>-Knead dough on a lightly floured surface until smooth and elastic, about 8 minutes.<br>-Place in a greased bowl, turning once to grease top.<br>-Cover and let rise in a warm place until doubled, about 1 hour.<br>-Punch down dough and divide into 12 pieces.<br>-Roll each piece into a 12-inch circle.<br>-Place on a greased baking sheet.<br>-Cover and let rise until doubled, about 1 hour.<br>-Preheat oven to 450 degrees f (230 degrees c).<br>-Bake in preheated oven until golden brown, about 10 minutes.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">7</th>\n",
       "      <td text-align=\"left\"><img src=\"https://www.moulinex-me.com/medias/?context=bWFzdGVyfHJvb3R8MTQzNTExfGltYWdlL2pwZWd8aDM2L2g1Mi8xMzA5NzI3MzI2MjExMC5qcGd8N2MxZDhmNmI5ZTgzZDZlZWQyZGQ4YjFlZjUyNDlkMTExYjdkZDdlZmFkY2I0N2NmNjljOGViNmExZjIyMDU4Yw\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Baked ziti</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/2/3/b/923b85bedb.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">cheese, pepper, onion, oil, tomato, salt, basil, clove</td>\n",
       "      <td text-align=\"left\">-Preheat oven to 350 degrees f.<br>-In a large bowl, combine the tomatoes, onion, garlic, basil, salt, pepper, and olive oil.<br>-Mix well.<br>-In a 9x13 baking dish, layer half of the ziti, half of the tomato mixture, half of the mozzarella, and half of the parmesan.<br>-Repeat layers.<br>-Bake for 30 minutes.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">8</th>\n",
       "      <td text-align=\"left\"><img src=\"https://hips.hearstapps.com/hmg-prod/images/delish-how-to-make-a-smoothie-horizontal-1542310071.png\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Berry banana smoothie</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/4/6/f/946f98b4de.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">yogurt, strawberries, banana, milk, honey, raspberries</td>\n",
       "      <td text-align=\"left\">-Place all ingredients in a blender and blend until smooth.<br>-Pour into glasses and serve.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">9</th>\n",
       "      <td text-align=\"left\"><img src=\"https://hips.hearstapps.com/hmg-prod/images/delish-how-to-make-a-smoothie-horizontal-1542310071.png\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Strawberry banana smoothie</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/d/d/0/9dd0f1d070.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">strawberries, milk, yogurt, ice, banana</td>\n",
       "      <td text-align=\"left\">-Place all ingredients in a blender and blend until smooth.<br>-Pour into glasses and enjoy !</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">10</th>\n",
       "      <td text-align=\"left\"><img src=\"https://www.simplyrecipes.com/thmb/45-YxKaW3K8CYhYzfxnYUDhz63Q=/1800x1012/smart/filters:no_upscale()/__opt__aboutcom__coeus__resources__content_migration__simply_recipes__uploads__2007__01__roast-chicken-carrots-sally-horiz-a-1800-51888dd5b8c44b6e9e656092f4f96f12.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Roasted chicken with thyme</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/9/1/6/99167224bf.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">pepper, chicken, salt, onion, oil, butter, garlic, thyme</td>\n",
       "      <td text-align=\"left\">-Preheat oven to 400 degrees f (200 degrees c).<br>-Place chicken in a 9x13 inch baking dish.<br>-Drizzle olive oil over chicken.<br>-Sprinkle with salt, pepper, garlic powder, onion powder, and thyme.<br>-Bake in preheated oven for 45 minutes, or until chicken is no longer pink in the center.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">11</th>\n",
       "      <td text-align=\"left\"><img src=\"https://www.simplyrecipes.com/thmb/45-YxKaW3K8CYhYzfxnYUDhz63Q=/1800x1012/smart/filters:no_upscale()/__opt__aboutcom__coeus__resources__content_migration__simply_recipes__uploads__2007__01__roast-chicken-carrots-sally-horiz-a-1800-51888dd5b8c44b6e9e656092f4f96f12.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Pork chops with onion and garlic</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/4/7/3/9473c8fca1.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">pepper, onion, oil, salt, pork, clove, flour, wine</td>\n",
       "      <td text-align=\"left\">-Season the pork chops with salt and pepper.<br>-Heat the oil in a large skillet over medium-high heat.<br>-Add the pork chops and cook until browned on both sides, about 5 minutes per side.<br>-Transfer the chops to a plate.<br>-Add the onion to the skillet and cook until softened, about 5 minutes.<br>-Add the garlic and cook until fragrant, about 30 seconds.<br>-Add the wine and cook until reduced by half, about 3 minutes.<br>-Return the pork chops to the skillet and cook until the pork is cooked through, about 3 minutes.<br>-Serve with the onion and garlic.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">12</th>\n",
       "      <td text-align=\"left\"><img src=\"https://www.jessicagavin.com/wp-content/uploads/2018/02/greek-salad-2-1200.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Black bean and corn salad</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/0/9/1/d/091dfc54c8.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">beans, corn, pepper, onion, cilantro, oil, avocado, cumin, juice</td>\n",
       "      <td text-align=\"left\">-Combine all ingredients in a large bowl.<br>-Toss gently to mix.<br>-Cover and chill at least 1 hour before serving.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">13</th>\n",
       "      <td text-align=\"left\"><img src=\"https://www.jessicagavin.com/wp-content/uploads/2018/02/greek-salad-2-1200.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Greek salad</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/3/c/3/93c36ab080.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">tomato, cheese, cucumber, olive, pepper, oil, onion, vinegar, salt, oregano, lettuce</td>\n",
       "      <td text-align=\"left\">-Combine first 6 ingredients in a large bowl.<br>-Add dressing; toss to coat.<br>-Sprinkle with cheese.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">14</th>\n",
       "      <td text-align=\"left\"><img src=\"https://www.twopeasandtheirpod.com/wp-content/uploads/2021/06/Greek-Quinoa-Salad4637.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Greek salad</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/8/4/0/9840aa9d24.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">onion, pepper, tomato, oil, salt, cucumber, vinegar, lettuce, cheese</td>\n",
       "      <td text-align=\"left\">-In a large bowl, combine the romaine, cucumber, tomatoes, onion, and feta cheese.<br>-In a small bowl, whisk together the olive oil, vinegar, salt and pepper.<br>-Pour over the salad and toss to coat.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">15</th>\n",
       "      <td text-align=\"left\"><img src=\"https://www.twopeasandtheirpod.com/wp-content/uploads/2021/06/Greek-Quinoa-Salad4637.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Mediterranean pasta salad</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/0/9/6/c/096c2c8ed8.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">olive, tomato, cheese, oil, pepper, onion, salt, capers, clove</td>\n",
       "      <td text-align=\"left\">-Cook pasta according to package directions.<br>-Drain and rinse with cold water.<br>-In a large bowl, combine pasta, tomatoes, olives, onion, capers, garlic, olive oil, salt and pepper.<br>-Toss to combine.<br>-Sprinkle with feta cheese.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">16</th>\n",
       "      <td text-align=\"left\"><img src=\"https://www.onceuponachef.com/images/2020/02/Double-Chocolate-Muffins.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Chocolate chip cookies</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/2/c/c/92cc1234a8.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">sugar, egg, flour, cocoa, salt, butter, chips, extract, baking_soda, oil, vanilla</td>\n",
       "      <td text-align=\"left\">-Preheat oven to 350 degrees.<br>-In a large bowl, cream together butter, oil, sugar, and vanilla.<br>-Add eggs and beat well.<br>-Combine flour, cocoa, baking soda, and salt; gradually add to creamed mixture.<br>-Stir in chocolate chips.<br>-Drop by rounded teaspoonfuls onto ungreased cookie sheets.<br>-Bake for 8-10 minutes.<br>-Cool on wire racks.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">17</th>\n",
       "      <td text-align=\"left\"><img src=\"https://www.onceuponachef.com/images/2020/02/Double-Chocolate-Muffins.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Chocolate chip muffins</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/a/c/2/9ac2fff987.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">sugar, flour, egg, chips, salt, baking_powder, baking_soda, extract, butter, milk, oil</td>\n",
       "      <td text-align=\"left\">-Preheat oven to 350 degrees f (175 degrees c).<br>-Grease 12 muffin cups or line with paper muffin liners.<br>-In a large bowl, cream together the butter and sugar until light and fluffy.<br>-Beat in the eggs one at a time, then stir in the vanilla.<br>-Combine the flour, baking powder, baking soda and salt; stir into the creamed mixture alternately with the milk.<br>-Fold in the chocolate chips.<br>-Spoon batter into the prepared muffin cups.<br>-Bake in the preheated oven for 20 to 25 minutes, or until a toothpick inserted into the center of a muffin comes out clean.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">18</th>\n",
       "      <td text-align=\"left\"><img src=\"https://hips.hearstapps.com/del.h-cdn.co/assets/17/31/1501791674-delish-chicken-curry-horizontal.jpg?crop=1.00xw:0.750xh;0,0.159xh&resize=640:*\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Spanish rice</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/b/f/2/9bf2d20349.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">rice, onion, pepper, tomato, clove, oil, salt</td>\n",
       "      <td text-align=\"left\">-Heat oil in a large skillet over medium heat.<br>-Add onion and garlic and saute until tender.<br>-Add rice and stir until rice is coated with oil.<br>-Add tomatoes, salt and pepper.<br>-Bring to a boil.<br>-Reduce heat to low, cover and simmer for 20 minutes.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th text-align=\"left\">19</th>\n",
       "      <td text-align=\"left\"><img src=\"https://hips.hearstapps.com/del.h-cdn.co/assets/17/31/1501791674-delish-chicken-curry-horizontal.jpg?crop=1.00xw:0.750xh;0,0.159xh&resize=640:*\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">Shrimp creole</td>\n",
       "      <td text-align=\"left\"><img src=\"data/demo_imgs/test/9/e/8/c/9e8c9706ea.jpg\" width=\"200\"/></td>\n",
       "      <td text-align=\"left\">shrimp, pepper, onion, clove, oil, tomato, salt, rice, celery, broth, juice</td>\n",
       "      <td text-align=\"left\">-Heat oil in a large skillet over medium-high heat.<br>-Add onion, bell pepper, celery, and garlic; saute 5 minutes or until tender.<br>-Add tomatoes, broth, salt, and pepper; bring to a boil.<br>-Stir in shrimp; cover, reduce heat, and simmer 5 minutes or until shrimp are done.<br>-Stir in lemon juice.<br>-Serve over rice.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table style=\"display:inline;\" width=\"100%\"></td></th>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Results\n",
    "instructions_display_cust = [instruction.replace('\\n', '<br>') for instruction in instructions_display_cust]\n",
    "\n",
    "df_r = pd.DataFrame({'query_image': [entry for entry in demo_urls for _ in range(2)],\n",
    "                     'similar recipe': titles_display_cust, \n",
    "                     'recipe_image': images_display_cust,\n",
    "                     'ingredients': ingredients_display_cust, \n",
    "                     'instructions': instructions_display_cust})\n",
    "\n",
    "# Show results\n",
    "display_table(df_r, titles=['Results'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98f561df",
   "metadata": {},
   "source": [
    "Even though embeddings coming from Facebook's model were slightly better, this pretrained model gave us pretty reasonable results and with some fine-tuning, it could work just as well."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "057d3e01",
   "metadata": {},
   "source": [
    "The advantage of using smaller embeddings is the cost of storing these embeddings and the time needed to create them and insert them. The disadvantage is that - the smaller the embeddings, the less information they hold."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2685b55",
   "metadata": {},
   "source": [
    "## Delete the index"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b64d2375",
   "metadata": {},
   "source": [
    "Delete the index once you are sure that you do not want to use it anymore. Once it is deleted, you cannot reuse it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4743376f",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%script echo skipping\n",
    "pc.delete_index(index_name)\n",
    "pc.delete_index(index_name_cust)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
